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Capacity of electron-based communication over 
bacterial cables: the full-CSI case 

Nicol5 Michelusi and Urbashi Mitra 


Abstract 

Motivated by recent discoveries of microbial communities that transfer electrons across centimeter- 
length scales, this paper studies the information capacity of bacterial cables via electron transfer, which 
coexists with molecular communications, under the assumption of full causal channel state information 
(CSl). The bacterial cable is modeled as an electron queue that transfers electrons from the encoder 
at the electron donor source, which controls the desired input electron intensity, to the decoder at the 
electron acceptor sink. Clogging due to local ATP saturation along the cable is modeled. A discrete-time 
scheme is investigated, enabling the computation of an achievable rate. The regime of asymptotically 
small time-slot duration is analyzed, and the optimality of binary input distributions is proved, i.e., the 
encoder transmits at either maximum or minimum intensity, as dictated by the physical constraints of 
the cable. A dynamic programming formulation of the capacity is proposed, and the optimal binary 
signaling is determined via policy iteration. It is proved that the optimal signaling has smaller intensity 
than that given by the myopic policy, which greedily maximizes the instantaneous information rate but 
neglects its effect on the steady-state cable distribution. In contrast, the optimal scheme balances the 
tension between achieving high instantaneous information rate, and inducing a favorable steady-state 
distribution, such that those states characterized by high information rates are visited more frequently, 
thus revealing the importance of CSI. This work represents a first contribution towards the design of 
electron signaling schemes in complex microbial structures, e.g., bacterial cables and biofilms, where the 
tension between maximizing the transfer of information and guaranteeing the well-being of the overall 
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bacterial community arises, and motivates further research on the design of more practical schemes, 
where CSI is only partially available. 


I. Introduction 

Cellular respiration relies on a continuous flow of electrons from an electron donor (ED) to 
an electron acceptor (EA) through the electron transport chain (ETC) of the cell to produce 
energy in the form of the molecule adenosine triphosphate (ATP), and to sustain vital operations 
and functions 0 Section 16.2]. This fundamental mechanism is well-known for individual, 
isolated cells. However, in the past decade, remarkable discoveries of multi-cellular microbial 
communities that transfer electrons between cells and across multi-cellular structures have been 
made 0. This mechanism, termed electron transfer, has been observed in molecular assemblies 
known as bacterial nanowires, and in macroscopic architectures, such as biofilms and multi¬ 
cellular bacterial cables 0. These experimental observations raise the possibility of microbial 
communication via electron transfer, which coexists with the better-known communication strate¬ 
gies based on molecular diffusion 0, 0, 0, enabling cells to quickly sense and respond to 
their environment. 

In this paper, motivated by these experimental observations and building on our recent queuing 
theoretic model of bacterial cables 0. we study the capacity of bacterial cables under the 
assumption of perfect CSI at both the encoder and the decoder. The encoder controls the 
desired intensity of the electron signal entering the cable, and the decoder attempts to decode 
the transmitted message based on the measured output electron process. The bacterial cable 
(the channel) has an internal state evolving dynamically as a function of electrons leaving and 
entering the cable, and events occurring within the cable, and therefore falls within the broad 
class of channels with state, e.g., O-GD. Specifically, the cable is treated as a single black 
box, which takes electrons as input from the ED source and outputs electrons into the EA sink, 
and is modeled as a finite-state Markov channel GD. controlled by the input signal. That is, the 
detailed dynamics of electrons and of the states of the cells located along the cable, as discussed 
in 0, are not explicitly accounted for, but only global effects on the cable resulting from these 
local interactions are modeled. Electrons enter and exit the cable according to Poisson processes, 
whose intensities are functions of the internal cable state, which in turn determines the ability 
of the cable to relay electrons, and of the encoded signal. 
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The contributions of this paper are as follows. We characterize the capacity of a discrete¬ 
time version of the system, using results on the capacity of finite-state Markov channels with 
feedback derived in pT| , and prove the optimality of stationary Markov input distributions, which 
depend only on the current state of the cable. We then consider the regime of asymptotically 
small time-slot duration. Based on this asymptotic analysis, we prove the optimality of binary 
input distributions, thus extending previous results on the capacity of static Poisson channels 


1121 to our dynamic setting. We prove that the capacity maximization problem is a Markov 


decision process (MDP) p3| |, with state given by the internal cable state, action given by the 
expected desired input electron intensity, which generates the binary intensity signal, and reward 
given by the instantaneous mutual information rate, and can thus be solved efficiently using 


standard optimization algorithms, e.g., policy iteration (see [13|). We show that the optimal 
input distribution optimizes a trade-off between achieving high instantaneous information rate, 
and inducing an ’’optimal” steady-state distribution of the cable state, such that those states 
where the transmission of information is more favorable are visited more frequently. On the 
other hand, the optimal distribution for the myopic or greedy policy, which greedily maximizes 
the instantaneous information rate, without considering its impact on the steady-state distribution 
of the cable, performs poorly. In particular, we prove that the optimal expected desired input 
electron intensity is smaller than that dictated by the myopic strategy. In fact, larger input electron 
intensities tend to quickly recharge the electron reserves within the cable, hence the ATP reserves 
of the cells, resulting in a clogging of the cable rendering it unable to further relay electrons 
until its reserves discharge to sufficiently low levels. 

Remarkably, this work represents a first contribution towards the design of electron signaling 
schemes in complex microbial structures, e.g., bacterial cables and biofilms, where the tension 
between maximizing the transfer of information and guaranteeing the well-being of the overall 
bacterial community often arises, and thus motivates further research in this direction, e.g, 
using methods based on statistical physics 10. Moreover, our numerical evaluations reveal 
the importance of CSI, which enables adaptation of the input signal to the state of the cell, 
and thus motivates further research on the design of more practical schemes, where CSI is only 
partially available, and of state estimation techniques. 

Most of the recent literature on the design of biological communication systems is based 


on molecular dijfusion 0, [15|-|21|. The achievable capacity for the chemical channel is 
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investigated in p7|, under Brownian motion, and in [ 18|, under a diffusion ehannel. In Q, p^, 


novel moleeular modulations are proposed. In [20|, an in-vitro moleeular eommunieation system 


is designed and, in pT| , an energy model is proposed. In p2[ , upper bounds on the eapaeity 
of eommunieation networks over linear time-invariant Poisson ehannels have been investigated 
in the eontext of moleeular diffusion, based on the symmetrized Kullbaek-Leibler divergenee. 
Therein, a statie ehannel with inter-symbol interferenee is eonsidered, and, similar to our work, 
the optimality of binary input distributions is also proved based on sueh upper bound. In this 
paper, instead, we build on the stoehastie model reeently developed in [[8| to study the eapaeity 
of baeterial eables via electron transfer. The proposed ehannel model is dynamic, as opposed to 
static, and explieitly eaptures biologieal eonstraints of baeterial eables, whieh are not present in 
mierobial eommunieations based on moleeular diffusion, sueh as elogging of the eable indueed 
by loeal ATP saturation of the eells, resulting in degradation of the eleetron transfer effieieney 
of the eable, and the minimum input eleetron flow requirement in order to keep the eells alive. 

Finite-state Markov ehannels have reeeived signifieant attention in the information theoretie 
eommunity, e.g., 0-IID- The ease with feedbaek and CSI at both the eneoder and deeoder 
is eonsidered in HD- We speeialize the eapaeity formulation of pT| to our Poisson ehannel, 
and show that it ean be aehieved by input distributions independent of the feedbaek signal. The 
ease with no CSI has been eonsidered in [ (TOl and in 0, for finite-state Markov ehannels whose 
transition probabilities are independent of the input signal. The eapaeity of ehannels with i.i.d. 


states eontrolled by aetions seheduled by the eneoder has been eonsidered in [231, and in [24|, 
for the setting where aetions are generated adaptively. 


The eapaeity of eontinuous-time Poisson ehannels is known, and it has been derived in |12| 
for the static ease. The ease with i.i.d. bloek-fading, CSI at the reeeiver, and partial CSI at the 
transmitter, has been eonsidered in [[25|. The eapaeity of a Poisson ehannel with side information 


on spurious eounts generated by an adversary at the reeeiver is eonsidered in [26|. The baeterial 
eable eonsidered in this paper is also modeled as a eontinuous-time Poisson ehannel. However, 
unlike [ fT^ , [25|, [ |26| , the ehannel is finite-state Markov, and its state is eontrolled by the input 
signal generated by the eneoder. While the eapaeity of eontinuos-time ehannels is known, the 
eapaeity of diserete-time Poisson ehannels is unknown, and only upper and lower bounds have 
been derived [ |22l , [ |27l . 

This paper is organized as follows. In Seotion|I^ we present the system model and the diserete- 
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time representation. In Seetion III we analyze the eapaeity of the diserete-time model and study 


its asymptotie eapaeity. In Seetion]^ we present numerieal results. In Seetion [V] we eonelude 
the paper. The proofs of the theorems and propositions are provided in the Appendix. 

II. System model 

We eonsider a eontinuous-time eommunieation system based on a baeterial eable, depleted in 
Fig. HI A stoehastie model for a baeterial eable has been proposed in 0. The baeterial eable 
eontains C eells and has an internal state S{t) E S at time t, taking values from the state spaee 
S. S{t) evolves in a stoehastie fashion as a result of random events oeeurring within the eable, 
e.g., eleetrons entering (X{t)) and exiting (Y{t)) the eable at random times, as detailed in Q. 
The eommunieation system ineludes an eneoder, whieh maps the message m E {1,2,..., M} 
to a desired electron intensity X{t) E [Amin, Amax], t ^ [0,^], where T is the eodeword duration, 
and Amin > 0 and Amax > Amin are, respeetively, the minimum and maximum eleetron intensities 
allowed into the eable. These are physieal eonstraints indueed by the nature of the eable. We 


let p = 


A A„ 


< 1. Note that the stoehastie model developed in [8| assumes that the eells may 


die at random times, e.g., as a eonsequenee of an insuffioient input eleetron flow. In this paper, 
we assume that the minimum input eleetron intensity Amin is suffieient to keep the entire eable 
alive, so that eells do not diej^Note that Amin can also be interpreted as the dark current [ 12|. 
Eleetrons enter the baeterial eable following a Poisson proeess with rate \in{t)=A{S{t))\{t), 
where A(s) E [0,1] is the clogging state, and denotes the ability of the baeterial eable to aeeept 
eleetrons. A(s) is speeifie to our baeterial eable model, and is not present, e.g., in statie Poisson 


ehannels, where A(s) = 1 |12|. In partieular, y4(s)=0 if the baeterial eable is elogged and no 


eleetrons ean be aeeepted, and A(s)=l if all eleetrons ean be aeeepted without any loss. In 
general, A(s) takes value in [0,1], so that only a fraetion of the input eleetrons ean be aeeepted, 
depending on the state of the eable. For instanee, if the high energy external membrane of the 
first eell in the eable is full [j^, then no eleetrons ean be aeeepted and A(s) = 0 aeeordingly. The 
input eleetron proeess is modeled as a eounting proeess X{t) E (0,1, 2,... }, with X(0) = 0. 
Note that the eneoder does not have full eontrol of the timing and number of eleetrons released 


'The more general case where cells die at random times will he considered as future work, and can he analyzed using tools 
developed in (28). 
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Figure 1. Communication system over a bacterial cable. 


into the cable (X(t)), but only of X(t), so that X{t) is a random quantity, which depends both 
on the encoded signal X{t) and on the clogging state A{S{t)). The output electron flow is also 
modeled as a Poisson process with state-dependent intensity which is a function of 

the internal state of the cable. We let Y (t) G {0,1,2,...} be the counting process associated 
with the output electron flow, with Y (0) = 0, and we denote the maximum output intensity as 
fimax = max^/r(s). 

We assume that both the encoder and the decoder have full CSI, i.e., the state sequence 
5(0 t) = {S{t),t < t} is known at both the encoder and the decoder at time t. Therefore, the 
desired electron intensity at time t, X{t), is chosen as a function of the message m and the CSI 
5(0:f), X{t) = ft{m; 5(0:t))e[0, Amax]- On the other hand, the decoder, given the output electron 
counting process F(0:T)={V(f), fG[0, T]} and the cable state sequence 5(0:T), estimates the 
message m as m=g{Y{0:T), S{0:T)). 
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A. Bacterial cable model 

In this section, we describe a model for the bacterial cable state S (t). The model [[^ presumes 
that S{t) is given by the interconnection of the states of each cell in the cable, leading to high 
dimensionality. In fact, letting Sceii be the state space of the internal state of each cell, then 
the overall state space of the cable is 5 = 5^;;, which grows exponentially with the bacterial 
cable length C. Herein, we propose an approximation of the model presented in [[^, which treats 
the bacterial cable as a black box, and captures only the global effects on the electron transfer 
efficiency of the cable, resulting from local interactions of the cells along the cable. Specifically, 
we model S{t) as the number of electrons carried in the cable, i.e., the sum of the number of 
electrons carried in the external membrane of each cell, which participate in the ETC to produce 
ATP for the cell. The state space for this approximate model is 5 = {0,1,..., S'^ax}, where 5 'max 
is the electron carrying capacity of the bacterial cable. Letting be the electron carrying 

capacity of a single cell, we have that S'^ax = C ■ Smt’iP- The rationale behind this approximate 
model is as follows: when S{t) is large, the large number of electrons in the cable can sustain a 
large ATP production rate, so that the ATP pools of the cells are full, and the cable is clogged, 
resulting in A{S{t)) ~ 0. On the other hand, when S{t) is small, a weak electron flow occurs 
along the cable, so that the ATP pools are almost empty and the cells are energy-deprived, so 
that the cable can sustain a large input electron flow to recharge the ATP pools {A{S{t)) ~ 1). 

In order to capture this behavior, we make the following assumptions. 

Assumption 1 The clogging state A(s) is a non-increasing function of s, with A(5' ma x)=0. 
A(0)=1, A(s)>0, Vs<S'max- The output electron intensity, p{s), is such that p{0)=0, /i(s)>0, Vs>0. 

According to this simplified model, from state S{t) at time t, the state of the cable becomes 
= S{t) + 1 with rate Aj„(f), corresponding to the arrival of one electron into the cable, 
and to state = S{t) — 1 with rate p{S{t)), corresponding to one electron exiting the cable. 


B. Discrete-time model 

The bacterial cable can be modeled as a finite-state continuous-time Markov channel m 
with Poisson input and output. We now design a discrete-time system, enabling the computation 
of an achievable rate for the continuous-time system. In particular, we use an approach similar 


to [121, which focused on the capacity analysis of a static Poisson channel, and we extend it 
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to our finite-state Markov channel. We divide the codeword duration T into N slots of fixed 
duration A = T/N. The kth slot is the interval [kA, {k + 1)A), for fc G {0,1,..., iV — 1}. In 


[121, it is shown that capacity-achieving schemes for this discrete-time model asymptotically 
achieve the capacity of the continuous-time Poisson channel, for asymptotically small values 
of the slot-duration A —0. Similarly, we first design capacity-achieving schemes and the 
corresponding capacity for the discrete-time model, and then analyze the asymptotic regime 


A —)■ 0 in Section III The rationale behind this analysis is that, if 

A« ' 


/^max T 


( 1 ) 


then the probability that more than one event (i.e., multiple electrons entering/leaving the cable) 
occurs in a single slot is very small, of the order of o(A^), whereas the following events are most 
likely to occur: 1) no electrons enter/leave the cable, with probability ~ 1—A/i(s)—AA(s)A; 2) 
one electron enters the cable, with probability ~ AA(s)A; 3) one electron exits the cable, with 
probability ~ A/i(s). However, the following analysis holds for any A > 0. Let 


ak = X{{k + l)A)-X{kA), (2) 

(3kAY{{k + l)A)-Y{kA), (3) 


be the number of electrons entering and exiting the cable in slot k, respectively. We make the 


following assumptions, similar to |12|: 

1) X{t) is constant within each slot. We denote its constant value in slot k as A^, so that 
A(f)=Afc, Vf G [kA, (fc-|-l)A); 

2) The encoder and decoder, at the beginning of slot k, know the sampled CSI time-series 
S'q = (S'o, Si, , Sk), where Sk=S{kA), rather than the continuous time-series S{0:kA); 

3) Additionally, the receiver observes (3k. However, it assumes > 1 is a rare event. This 
event is indeed rare when A satisfies condition Q, since its probability is of the order of 
o(A^). Therefore, the receiver utilizes only the positive presence of electrons /3k = xWk > 
0) for decoding purposes, where x(-) is the indicator function, i.e., (3k = 0 ii no electrons 
are received in slot k, and (3k = 1 otherwise. 

We let, for a, b, > 0, 

A)=P(afe = a,(3k = h\Sk = s,\k = \). (4) 
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From the properties of Poisson proeesses, we have that 



1—A/r(s)—A74 (s)A+(7o,o(^5 s,^) a = 0,b = 0, 
AA(s)A -f Qi o(A, Sy A) o = 1,6 = 0, 

< 

A/i(s)+ 5 (o,i(A,s,A) a = 0,6 = 1, 

^ Qafii^ySyX) a > 0,6 > 0, 


(5) 


where 5 'a^b(A,s, A) are funetions sueh that ga^bi^, s, X) ~ o j'^max{a+6,2}^^ ^ (-q 

zero as when A —)■ 0, and = 0’ that p^^(a,6|s. A) is a 

probability distribution. The interpretation of Eq. © is that, when A satisfies the eondition 
in Eq. Q, the event (a, 6) = (0, 0) (no eleetrons enter/leave the eable) oeeurs with probability 
~ 1—A/r(s)—AA(s)A; the event (a, 6) = (1,0) (one eleetron enters the eable, no eleetrons leave 
the eable) oeeurs with probability ~ AA(s)A; whereas the event (a, 6) = (0,1) (no eleetrons 
enter the eable, one eleetron leaves the eable) oeeurs with probability ~ A/i(s); all the other 
events oeeur with probability of the order of o(A^). We let 


A) = P(S'fc+i = si\Sk = So, Afc = A). 


( 6 ) 


Sinee 5^+1 = Sk + ak — (3k, we have that 

OO 

A) =^ p^^^(a,So-Si + a|s, A), (7) 

a=(si-so) + 

and therefore, using Q, 

OO 

Ps ^(■So + l|so, A)=Ay4(s)A+( 7 a^(j_i(A, s. A), (8) 

a=l 

OO 

pf\so - l|so, A)=A/r(s)+^ 5 (a,a+i(A,s, A), (9) 

a=0 

OO 

Ps^^solso, X)=l-Ap{s)-AA{s)X+ ^ 5 'a,a(A, s. A), (10) 

a=0 

OO 

X) = ^ga,so-si+a{A, S, X), Vsi^{So-l, Sq, Sq + I}. (11) 

a=(si—so) + 

Einally, we define the joint probability of state transition Sk^Sk+i and ehannel output ^k as 

P^^lisi, 6|so, A)=P(S'fc+i=si, /3fc=6|S'fc=so, Afc=A). (12) 
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Using the fact that Sk+i = Sk + ak — f^k, we obtain 

0|so, A) = - So, 0|so, A)x(si > sq), 

CXD 

V^s)i^si,l\so,\) = ^ p^"^^(6+si-so,6|so, A). 

6=max{ So ~-s 1 ! 1} 

The encoding and decoding schemes for this discrete time model are defined as follows: 

• Encoding: in slot k, given the message m to be transmitted, and the CSI Sq, the encoder 
defines the desired electron intensity \k = fkim-jS^), taking values from the continuous 
alphabet [Amin,Amax]; 

• Decoding: given the CSI Sq and the output sequence $q~^, the decoder returns the message 
estimate m=g{SQ , ^q~^). 


Note that, similar to [12|, we have converted the original continuous-time Poisson channel 
into a discrete-time channel, with input AfcG[Amin, Amax] and binary output G {0,1}. However, 


unlike 1 1^ which focuses on a static Poisson channel, we now have a finite-state Markov 
channel, controlled by the input sequence, with transition probability P 5 ^^(si|so, \k) and output 
distribution 

P^sl{si,l\sQ,\k) 




P^^\si\so,Xk) 


Therefore, the optimal input distribution, which is analyzed in the next section, optimizes a 
trade-off between achieving high instantaneous information rate, and inducing transitions in the 
future to states characterized by large information rate. 


III. Capacity analysis 

We have the following proposition, which states an important property of the controlled 
Markov chain {{Sk,fik-i), k > 0}. 

Proposition 1 The Markov chain {{Sk, fik-i), k > 0}, with state space S x {0,1} and action 
space [Amin, Amax], is strongly irreducible and strongly aperiodic /[TT]/. 

The significance is that, if we view the Markov chain as a random walk on a directed graph, this 
directed graph is always strongly connected and all its states are of period one, irrespective of 


the input distribution generating {{Sk, fik-i)-, k > 0} [11|. In particular, under a stationary 
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Markov input distribution, whose optimality is proved in HD. the resulting Markov chain 
{(‘S'fc,/3fc_i), fc > 0} is stationary and ergodic. 

The capacity of finite-state Markov channels is studied in pT| , for the case where the encoder 
is provided with the output feedback 13k- Therein, it is proved that, for strongly irreducible and 
aperiodic Markov channels, the feedback capacity specialized to our case is given by 

‘S'max 

Q = max^ ^ 'K[^\s,h)ll^\s,h), (13) 

s=o be{o,i} 

and is achieved by a stationary Markov input distribution (the optimizer of ([T3|)), which 
maps the current state {Sk, (dk-i) = (s,&) to a probability distribution over the input signal, 
i/(A|s,6). The terms 7ri^\s,b) and are, respectively, the steady-state distribution and 

the instantaneous mutual information rate of state {Sk,f3k-i) = (s,&), induced by the input 
distribution u. li^\s,b) is defined as 

li^\s,b) = ^I{Xk-Jk,Sk+i\Sk = s,Pk-i = b,u{-\s,b)). (14) 


Note that, in general, the input signal distribution i/ is a function of the current state Sk = s and 
of the feedback signal /3k-i = b. However, the feedback signal ^k-i = b is not provided to the 
encoder in our model. In the following proposition, we show that the optimal input distribution 


solution of (13), denoted by is, in fact, independent of (3k-i. 


Proposition 2 The optimal input distribution maximizing the capacity C\ in ( [7^ , is such 
that 


0) = 1), Vs G S, (15) 

i.e., it is independent of the feedback signal (3k-i = b. Under such a distribution, the capacity 
is given by 

'S'max 

Q = max ^ 7r^^)(s)/(^)(s), (16) 

where the optimization is over the set of stationary Markov input distributions v which map the 
current cable state Sk = s to a probability distribution over the input signal, i/(A|s). The terms 
7ri^\s) and li^\s) are, respectively, the steady-state distribution and the mutual information 
rate in state Sk=s, induced by v. Iy^\s) is defined as 

= ^I{Xufik,Sk+i\Sk = s,i/(-|s)). (17) 
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Proof: See Appendix A. ■ 

That a distribution z 2 (-|s) independent of the feedbaek f3k-i is optimal ean be intuitively 
explained: the state Sk is an information state, and eaptures all information about the past, so that 
knowledge of is irrelevant; moreover, sinee the instantaneous information rate Iu‘^\Sk, h-i) 
is a eoneave funetion of the input distribution, it is maximized by the input distribution u{-\Sk) 
independent of the feedbaek signal Pk-i, rather than by randomizing between z/(-|S'/co) and 
o{-\Sk, 1), depending on the value of f3k-i ^ {0,1}. The implieation is that the eapaeity (7^ in 


Eq. ( |13| ) is aehievable only with the CSI Sk available at the eneoder, but without feedbaek 
as it is assumed in our model. 

In this paper, we are interested in the analysis of the asymptotie regime A —0, thus extending 


the eapaeity of static Poisson ehannels pA| | to our dynamic setting. The asymptotie eapaeity, 
defined as 


C* = lim Cl, 

A-s-O 


is eharaeterized in Proposition 


Proposition 3 The asymptotic capacity C* is given by 

'S'max 

C* = max 'Ky{s)Iy{s), 


s=0 


where we have defined the asymptotic mutual information rate as 

Vim iy\s)=A{s) / z/(A|s)Alog 2 ( 1 dA, 


A-s>0 


'K 


A(s) 


and the asymptotic steady-state distribution as 


s-l 


VTi 


(s) = \im 7ry\s) = JJ 


j=o 


dU + 1 ) 


where 


(18) 


(19) 


( 20 ) 


( 21 ) 


7rx(0) = lim 7rf,^^(0) =--- 

^ ^ ^t=l 1 li=0 ^l{j+l) 

and we have defined the average desired input electron intensity 


( 22 ) 


— / i^(A|s)AdA. 

^ -^min 


(23) 


Proof: See Appendix B. 
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Note that the steady-state distribution ( |2T| ) is a funetion of A only. Sinee we are eonsidering 
infinitely small time-slot durations A —0, the input signal {A^} averages out over short time 
intervals, henee only its expeeted value A(s) affeets state transitions, rather than its speeifie 
distribution. 

The eapaeity optimization problem in Eq. ( [T9l ) highlights the following trade-off: the optimal 
input signal should, on the one hand, aehieve high instantaneous information rate Iu{Sk) (e.g., by 
employing the myopie distribution zz(-|s) = argmax Vs); on the other hand, it should be 

designed in sueh a way as to favor the oeeurrenee of states eharaeterized by large instantaneous 
information rate. These two goals are in tension. In faet, the instantaneous information rate 
is maximum in states with large elogging state A(s) ~ 1, i.e., when Sk is small and the 
baeterial eable is deprived of eleetrons. Visits to these states are aehieved more frequently by 
ehoosing Xk = Amin with probability one. However, under this deterministie input distribution 
the instantaneous information rate is zero. The optimization problem in ( [T9] ) ean be interpreted 
as a MDP, with state spaee S, aetion zz(-|s) in eaeh state (eaeh aetion is a probability distribution 
over the input signal A^ G [Amin, Amax]), the reward r(i/(-|s),s) under aetion in state s is 

given by 


r(z/(-|s), s) = A(s) 


z/(A|s)Alog 2 


A 


A(s) 


dA, 


(24) 


and transition probability from state Sk = sq to Sk+i = si under aetion zz(-|so) given by 


P(S'fc+i = si\Sk = so,z/(-|so)) = < 


1 - 5/i(so) - ()A(so)A(so) Si = So, 

<5A(so)A(so) Si = So + 1, 

(5^(so) Si So 1, 

0 otherwise, 


(25) 


where A(s) is defined in (23) and 6 is any eonstant satisfying 


6 < 


/^max T Aj] 


(26) 


in order to guarantee a feasible transition probability matrix. In faet, by solving the steady-state 


equations with transition probabilities in Eq. (25), we obtain the steady-state distribution 7r);(s) 

are equivalent to 


stated in the proposition. Note that the transition probabilities in Eq. 

E[pg^(si|so, A)] (see Eq. (8)), where the expeetation is with respeet to A ~ i/(A|so)), but without 
the terms ga,biS, so. A), whieh are negligible in the asymptotie regime A —)■ 0. 
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Therefore, the optimal input distribution which maximizes the capacity in Eq. (19), denoted 


by I/*('Is), can be determined by using standard MDP algorithms, such as policy iteration [13|. 
Note that, under any v, the steady-state distribution of Sk is a function of the average desired 
input electron intensity A only. Given A, there exist an infinite set of distributions v which induce 
the same average desired input electron intensity A and the same steady-state distribution vr^. 
Mathematically, this set is defined as 


V(A) 


i/(A|s)AdA = A(s), Vs G 5 


(27) 


Therefore, we have a degree of freedom in optimizing z/ over such set V(A), for every possible 
choice of A. Since all u G V(A) induce the same steady-state distribution, this optimization 
is equivalent to maximizing the instantaneous mutual information rate /^(s) in each state. 
Intuitively, /j/(s) is maximized by choosing input symbols A^ which can be distinguished more 
clearly at the receiver and are maximally different, i.e., \k G {Amin, Amax}- This is formalized 
in the following proposition, which proves the optimality of binary input distributions. 


Proposition 4 The optimal input distribution has the form 


Z^*(Amax|s)=P(Afe = Amax|5'fc = s) = 
«^*(Amin|s)=P(Afc = XminlSk = s) = 


A*(s)-A, 


A. 


A. 


Amax- A* (S) 


(28) 

A - A ’ 

/'max /'mm 

where A*(s) is the optimal expected desired input electron intensity in state s, defined as the 
maximizer of the capacity, 


C* = _ max ^ 7rx(s)/(A(s),s), 

A:4Si-^[Aniin ,Amax] 


S=0 


where I{x, s) is given by, for x G [Amin, Amax], 

Arna.",; 


I{x, s)=y4(s) 

Proof: See Appendix C. 


a; logs 


X 


P(Amax ai) , f 

-^- log2 - 

1-p \p 


(30) 


(31) 


A similar result has been proved in [12| for the case of a static Poisson channel. Therein, 


the optimal input distribution is the myopic input distribution, studied in Section III-A, which 
maximizes the instantaneous mutual information rate, i.e.. 


\mp — argmax I{x,s), Vs G 5. 

^ ^ [Amin > Amax ] 


(32) 
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(Note from ( |^ that \mp is independent of s; the dependenee on s has thus been removed 
aeeordingly). In faet, for a statie Poisson ehannel, the tension between maximizing the instanta¬ 
neous mutual information rate and indueing a favorable steady-state distribution does not arise. 
Proposition 1^ thus represents the extension of [|T2| to finite-state Markov ehannels. 

From Proposition it follows that the optimization ean be restrieted only to binary input 
distributions, whieh alloeate non-zero probability only to the minimum and maximum input 
eleetron intensities, and zero probability to any intermediate values. Equivalently, only the 


expeeted eleetron intensity A(s) needs to be optimized in eaeh state, as the maximizer of (30), 


using optimization tools sueh as poliey iteration [13|. 

Note that the myopie poliey, defined in (32) and studied in Seetion III-A[ maximizes the 
instantaneous mutual information rate I{x, s). However, in doing so, the myopie poliey negleets 
the steady-state behavior of the eable, and may thus induee frequent visits to states eharaeterized 
by small elogging state A{Sk) — 0, i.e., when Sk approaehes S'max- On the other hand, the 
optimal input distribution A*(s) balanees this tension by giving up part of the instantaneous 
transfer rate in favor of a better steady-state distribution in the future, i.e., states eharaeterized 
by large elogging state A{Sk) — 1 where a larger instantaneous mutual information rate ean be 
aehieved. Note that any input distribution A(s) larger than the myopie one Amp is deleterious to 
the eapaeity for the following two reasons: 1) a lower instantaneous mutual information rate is 
aehieved, eompared to Amp (by definition of the myopie distribution, whieh maximizes I(x, s)); 
2) faster reeharges of eleetrons within the eable are indueed, resulting in frequent elogging of 
the eable, where the instantaneous mutual information rate is small. This property is formalized 
in the following theorem, whieh proves a struetural property of A*(s), by eomparing it with the 
myopie distribution. 

Theorem 1 The optimal signal distribution \* is such that A*(s) < Xmpi'^s. 

Proof: See Appendix D. ■ 

Theorem ean be exploited in the eapaeity optimization problem ( [^ , by restrieting the 
optimization only to distributions sueh that A(s) e [Amin, Amp]- We now present the poliey 


iteration algorithm [13|, whieh exploits this faet. 


Algorithm 1 (Policy iteration algorithm) 1) Initialization.- initial average desired electron 
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intensity i = 0; e > 0 

2) Stage i, policy evaluation step; compute the achievable rate under policy A^, 

‘S'max 

s=0 


using (21) and (31 ). Let Zi)W(O) = 0, and recursively, for s G {1, 2,..., S'max}, 


- l)DH(g - 1) + - /(AH(g - 1), g - 1) 

A(s — 1 )AW(s — 1) 


(33) 


(34) 


A['+^1(s) = < 


3) Stage i, policy improvement step;/or each s e S, determine a new policy as follows: 

\mp iAW(s + 1) > 0, 

Amin + 1) < log 2 (e) + log 2 (p), (35) 

otherwise. 

4) Convergence test; — (7^ < e, return otherwise, let i := i + I and repeat 

from step 2). 

The following proposition states the optimality of Algorithm 
Proposition 5 Algorithm determines the optimal policy X* when e —)■ 0. 


Proof: The optimality of the policy iteration algorithm is proved in [ 131. The specific forms 


of the policy evaluation and improvement steps, which exploit the structure of our model, are 
proved in Appendix E. ■ 


The algorithm can be initialized with the myopic policy, studied in Section ITT-A 


A. Myopic codebook generation 


The myopic policy, defined in (32), is the expected desired input electron intensity which 
maximizes the instantaneous mutual information rate. Since the mutual information rate I{x,s) 
is a concave function of x, with /'(Amin, s) > 0 and /'(Amax,s) < 0, it is maximized by the 
unique x G (Amin, Amax) such that I'{x,s) = 0. From ( [^ , it is given by 

A. 


Xmp — 


'max — —^ 

i-p 


-p 


(36) 


and is independent of s (the argument s has been removed accordingly). The resulting achievable 
rate under this myopic input distribution, denoted by Cmp, is given by 

Cmp = AmpX max 


-p log2(e) + log2(p) 
e 1 ~ P 


(37) 
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where Amp = '^Mpi-s)Ais) is the average elogging state and ttmp{s) is the steady-state dis¬ 
tribution under the myopie poliey. Note that Xmp only maximizes the instantaneous information 
rate, without taking into aeeount its effeet on the steady-state distribution, so that the resulting 
average elogging state Amp, and the eapaeity Cmp, may be small. 


IV. Numerical Results 

In this seetion, we present numerieal results. We eonsider a eable with eleetron eapaeity 
‘S'max = 1000. The elogging state 74(s) and output rate /i(s) are given 

s 


A(s) = x(s < 5'max) 

/i(s) = 0.6 -f 0.8 


1 ~ (1 ~ ^min) 






(38) 

(39) 


where A^am is the minimum clogging state value, taking values in the range [0,1]. Note that A(s) 
is peeuliar to the baeterial eable, sinee it eaptures saturation effeets oeeurring loeally within eaeh 
eell, and eausing the overall eable to elog. 

In Fig. 1^ we evaluate the aehievable information rate under the optimal input distribution 


(OPT), eomputed with the poliey iteration algorithm [13|, and the myopie input distribution 


(MP). We plot the aehievable information rate as a funetion of Amin G [0,1]- We note that 
the aehievable information rate inereases with Amin for both sehemes. This is beeause, as Amin 
inereases, both A(s) and the instantaneous information rate I{x,s) inerease as well (see Eqs. 
(381 and (|3T])). Intuitively, the larger A(s), the better the ability of the baeterial eable to transport 
eleetrons. 

OPT outperforms MP by ~ 9% for small values of Amin- This can be explained with the help 
of Fig. for the ease Amin=0. In this ease, elogging is severe when the eable state approaehes 
the maximum value S'max- For instanee, if Sk > *S'max/2, then A(S'fc) < 0.5 and the rate of 
eleetrons entering the eable is less than halved, resulting in low instantaneous information rate. 
Therefore, in order to aehieve high instantaneous information rate, the state of the eable Sk 
should be kept small, e.g., below S'max/2. MP greedily maximizes the instantaneous information 
rate, but this aetion results in an unfavorable steady-state distribution, sueh that the eable is often 


^The specific choices of A(s) and jj.{s) have been discussed with Prof. M. Y. El-Naggar and S. Pirbadian, Department of 
Physics and Astronomy, University of Southern California, Los Angeles, USA. 
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Figure 2. Achievable information rate as a function of the minimum clogging state value, Amin, for the MP and OPT input 
distributions. 


in large queue states Sk > 5'max/2, where the elogging is signifieant (A{Sk) < 0.5) and most 
electrons are discarded at the cable input. On the other hand, OPT gives up some instantaneous 
information rate in order to favor the occupancy of low queue states, where 74(s) approaches 
one and the transfer of information is maximum. Note that MP does not require CSI at the 
encoder, as seen in (j^. Therefore, Fig. demonstrates the importance of CSI, which enables 
the adaptation of the input signal based on the state of the cable. Finally, note that OPT and MP 
approach the same value of the average information rate when Amin—)-l. In fact, in this case the 
instantaneous information rate /(x,s) is the same in all states, except S'max, where A(S'inax)=0 
and /(x, S'max)=0. However, state S'max is visited very infrequently, resulting in a negligible 
degradation of MP with respect to OPT. 
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Figure 3. Cumulative steady-state distribution (SSD) and instantaneous information rate for the MP and OPT input distributions; 

^min = 0. 


V. Conclusions 

In this paper, we have studied the eapaeity of baeterial eables via eleetron transfer, for 
the case where both the encoder and the decoder have full causal CSI. We have studied a 
discrete-time version of the system, which enables the computation of an achievable rate for the 
continuous-time system, based on known results on the capacity of finite-state Markov channels. 
We have analyzed the regime of asymptotically small time-slot duration, based on which we 
have established the optimality of binary Markov input distributions, which are functions of the 
current state only. We have shown that the optimal distribution optimally balances the tension 
between optimizing the instantaneous mutual information rate, and inducing a favorable steady- 
state distribution, e.g., to states characterized by large clogging state, and we have proved that 
it is smaller than that given by the myopic policy, which greedily maximizes the instantaneous 
information rate neglecting its effect on the steady-state distribution of the cable. We have 
shown that the optimal probability that generates the binary intensity signal as a function of the 
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current cable state can be determined efficiently via the policy iteration algorithm. Our numerical 
evaluations reveal the importance of CSI, which enables adaptation of the input signal to the 
state of the cell, and thus motivates further research on the design of more practical schemes, 
where CSI is only partially available, and of state estimation techniques. 

This work represents a first contribution towards the design of electron signaling schemes 
in complex microbial structures, e.g., bacterial cables and biofilms, where the tension between 
maximizing the transfer of information and guaranteeing the well-being of the overall community 
arises, and thus motivates further research in the design of signaling schemes in large-scale 


microbial systems and bio-films, e.g, using methods based on statistical physics [14|. 
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Appendix A: Proof of Proposition [2] 

Let 1 / be a stationary Markov input distribution with feedback, which maps the current state 
{Sk, = (s, h) to a probability distribution over the input signal, z/(A|s, h), but is independent 


of the past [ |ll| . From Proposition [ij the state sequence {(S'^,/c > 0} is a strongly 
irreducible and aperiodic Markov chain, controlled by z/. Given {Sk-i,l3k-2) = {s'ih'), which 
occurs with steady-state probability the system moves to state {Sk,j3k-i) = (s,6) 

with probability h')p^^{{s, 6|s',A)dA, independent of the past. Therefore, the steady- 

state distribution of {Sk, (3k-i) induced by the stationary Markov input distribution v is the unique 
solution of the system of steady-state equations 


TT 


(A) 


(s,&) = ,h') 


z/(A|s', b')p^^{{s, 6|s',A)dA, Vs, b. 


(40) 


s' ,b' 


We now define a new stationary input distribution without feedback, denoted by u. Let, for 
each s e S, 


i>(A|s) = 


^^7rl^^(s,6)z/(A|s,6) 




s) 


, VA, 


(41) 
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where = 7ri‘^\s,0) + 7ri‘^\s, 1). The steady-state distribution of (Sk,/3k-i) indueed by 

i> is the unique solution of the system of equations 


b)= ^ b')I b\s', A)dA. 

g/ y ^ '^min 

.(A). 


-An 


s')p%i 
AA)/„ - ^(A)/ 


(42) 


We now show that vri- '^(s, b) solves ^2), henee tt) "^(s, b) = ttI b), Vs, b, i.e., we prove that 

^-^max 

'^i'^\s,b)=^7rl^\s',b') i>{X\s')p^^l{s,b\s',X)dX. (43) 

g/ y ^ -^min 

In faet, substituting ( |4T] ) into ( [43] ), we obtain 

-(^)(s,6) = 5^vr(^)(s',6') r 


vr,\ 


Ey7r.(s',2/)z/(A|s',?/) , 

pU(s,&|s ,A)dA 


s’,bi 




= 2/) 


^a(A|s',|/)p^'^](s,6|s', A)dA, 


(44) 


s',y 


and this holds true from (40). Therefore, the two polieies z/ and z> aehieve the same steady-state 
distribution 7ri^\s,b) = 7rl^\s,b), ys,b. 

The mutual information rate li^\s,b) is given by 


s',b' 

where from ( [T?] ) we have defined 


(45) 


Z^^\x,s,s',b') = - — 


x{X)pfl{s\b'\s,X) log2 


'/A::^(^>iS(^'’^>’^')dA'' 


P'sl{sXb'\s,X) 


dA. 


(46) 


It ean be shown that s, s',b') is a strietly eoneave funetion of the input distribution x. 

Therefore, using ( |45| ) we obtain 

fee{o,i} s',b' fee{o,i} 


be{o,i} V'®! 


_ ^(A) 


4^ (s)5^^^^H^(-k),s,s',6') = 7r44)4^ (s), 

s',b' 
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with equality if and only if i/(-|s,6) = z/(-|s),V6 G {0,1}, where I^^\s) is given by (17). In the 


last step, we have used the definition of u in (41) and the faet that TtI ^(s) = TTp -^(s). Finally, 
we obtain 

‘S'max 5'max ‘S'max 

E E 7r^(.s,b)I^(s,b) = E E 7r£;(s,6)4(s,6) < ^ 7r£;(s)/£;(s), (47) 


s=o be{o,i} 


s=o 6e{o,i} 


s=0 


with equality if and only if F(A|s) = i/(A|s, 6), VA, s, 6, so that the input distribution 0 aehieves 
a larger average information rate than the original input distribution ly. The proposition is thus 
proved. 


Appendix B: Proof of Proposition [3] 


The mutual information rate li^\s) is defined in ( |17[ ) and is given by 

s, s', 6'), 

s',b' 


(48) 


where ,b') is defined in (46). In partieular, for (s', 6') ^ {(s, 0), (s + 1, 0), (s— 

1,1)}, from ([^ when A —)■ 0, we have that 

lim Z(^)(i/(-|s),s,s',6') = 0, (49) 


A-s>0 


sinee, in this ease, P^^^^{s\b'\s,\) ~ o(A^). Therefore, it follows that 


Im/,^^^(s) = fim^ [E^^^(z/(-|s), s, s, 0) 


A-5-O 


+Z^^\v{-\s), s, s + 1, 0) + Z^^\v{-\s), s, s - 1,1)] . 


(50) 


By negleeting the probability terms of order o(A^) in ([^-(11), from (46) we thus obtain 


Z(^)(i/(-|s),s,s,0)~-y x(A) -/i(s)-A(s)Aj logs 
- log 2 (e) / x(A)A(s)(A - A(s))dA = 0, 


1/A — /i(s) — A(s)A(s) 
1/A — /i(s) — A(s)A 


dA, 


(51) 


s, s + 1,0)= / x(A)A(s)A logs 

'^min 

Z(^)(i/(-|s),s,s - 1,1) ~ 0, 


A 


A(s) 


dA, 


(52) 


where, in ( [ST] ), we have applied L’HopitaPs rule to eompute the limit. Finally, using (48), we 
obtain limA-s-o= Iu{s) as given by (20). 
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We now compute the asymptotic steady-state distribution. The steady-state distribution ti, 
is the unique solution of the balance equations, for s G {1,2,..., S'max}, 

Sjnax ^Amax 

/ jy{X\s')p^s^\s"\s',X)dX 


(A) 


s '=0 


off — c '^min 


g 1 /-Aniax 

/ iy{X\s')p^s^\s"\s',X)dX. 

s'=s s"=0 ' 


(53) 


By reordering the terms, we can rewrite 

7r^^^(s —1) / z/(A|s — (s|s — 1, A)dA — 7r^‘^^(s) / z/(A|s)p^^ (s — l|s, A)dA (54) 




‘S'max /»An 


= vrl^^fs)/ z/(A|s)p^^ (s"|s, A)dA — 7r^^^(s — 1) / z/(A|s')pg^ (s"|s', A)dA 


<j"=0 

S—1 

/ 


XI / ^(^|5')p5fHs"k',A)dA - X / i/(A|s')plfHs"|s',A)dA. 


s" = s+l “'-''min 
5 2 ‘S'max /‘An 


'3 


s'=s-l-l 


S"=0 


s'=0 


o>> — Q ^ '^min 


Using ([^-([TT]) and the fact that the terms ga,bi^, s, X) are of the order of it can 


be shown that the right hand-side of (54) contains terms of the order of o(A^). Therefore, using 
(|^-([T^ to express the left-hand side and including the terms o(A^) in the right-hand side, (54) 
becomes 


7r^^^(s —1) / z/(A|s — l)AA(s — l)AdA — 7r^^^(s)A/i(s) = o(A^), 

^ -^min 

or equivalently, by dividing each side by A, 

- l)A((s - l)A(s - 1) - Trl^\s)p{s) = o(A). 

Therefore, letting A —)■ 0, we obtain 

A(s-ms-i) 

/i(s) 

The steady-state distribution ([2T]) is obtained by solving recursively, thus proving the proposition. 


(55) 


(56) 


(57) 


Appendix C: Proof of Proposition |4] 


Using ( [27] ), the capacity C* can be decoupled as 

'Smax 


C* = max max > Tri{s)I,y{s), 

A ‘^eV(A)^ 


(58) 
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where the inner optimization is over the set of distribution with a predefined average desired 
input eleetron intensity profile A, and the outer optimization maximizes over sueh A. 

We now solve the inner optimization problem 


max 7ii{s)I„{s). 

-6V(A) ^ 


(59) 


Sinee I^is) is only a funetion of and is independent of Vs' ^ s, and the steady-state 

distribution is the same for all z/ G V(A), we obtain 


‘S'max 'S'max 

max V7rx(s)J^(s) = V 7rx(s)/(A(s), s), 


where we have defined 


(60) 


/(A(s),s) = max/j,(s), s.t. / z/(A|s)AdA = A(s). 


Equivalently, using (20), 


_ ^Amax / \ 

/(A(s),s) = maxv4(s) / a;(A)Aloga ( ) dA, 


"An 


r-An 


'^max / -^max 

s.t. I a:(A)AdA = A(s), / a:(A)dA = 1. 

^Amin ^Amin 


Letting 


z{y) =y\og 2 


y 




the optimization problem ([6^ is equivalent to 


^ Amax 

/*(s) = maxy4(s) / a;(A) 2 ;(A)dA 

^Ajnax ^Amax 

s.t. / a;(A)AdA = A(s), / a;(A)dA = 1. 


* An 


'An 


Note that z{y) is a eonvex funetion of y. Therefore, we have that 

f \\\\A \ ^ \ I Nj 

/ x(A)z(A)dA^-- - ^(Aniin)“l“T 7 2;(Aiiiax)- 


Amax Afniji 


A max A mm 


( 61 ) 


(62) 


(63) 


(64) 


Therefore, the maximum in (62) is attained by a distribution u* whieh seleets A^ G {Amin, Amax} 


with probabilities given by (28), so as to attain the eonstraint E[Afc|S'fc = s] = A(s),Vs. Under 
the optimal distribution, the expression of /(x, s) ean be shown to be as in ( [sT] ). The proposition 
is thus proved. 
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Appendix D: Proof of Theorem [T] 


We prove the theorem by analyzing struetural properties of the n-step eost-to-go funetion p3 | 
from state s, denoted by Vn{s). Using (25) and the struetural properties of Proposition whieh 
restricts the input signal to binary values Xk G {Amin, Amax}, Vn{s) solves recursively 


Vn{s) = max {J(x, s) + 5A(s)xl4-i(s + 1) + (i/i(s)14-i(s — 1) 

['^mirn'^max] 

+(1 - 5A(s)a; - 5/i(s))14_i(s)} , 


(65) 


where Vo(s) = 0, Vs. The maximizer of (65) is the optimal average desired input electron 
intensity in step n, denoted by A„(s), yielding 

Vnis) = 14 ._l(s) + l(Xn{s), s) — 5A(s)A„(s) [14i-l(s) — Vn-l{s + 1)] 

+ 5/i(s)[K_i(s - 1) - K_i(s)]. (66) 

We prove that 

Amin ^ A^(s) ^ Ajvfp, Vs, Vu. (67) 

Then, since An(s) converges to the optimal policy as n —>■ cx), i.e., Ari(s) —)■ A*(s) for n —)■ cx) 


1131, it follows that the condition (67) implies Amin < A*(s) < Xmp by taking the limit n —)■ oo, 
thus proving the theorem. 

It can be shown that I{x,s), given by ( [31] ), is a strictly concave function of x, with first 
derivative 


, ^ d/(a;,s) 

^ ^ da; = 


log2 


max \ P ^ / * 

, + q- log2 - 

ex 


1-p 


( 68 ) 


Therefore, given 14_i, the objective function in ( [65] ) is a concave function of x. Its derivative 
with respect to x is given by 


f{x) = I'{x,s) - M(s)[I4_i(s) - I4-l(s + 1)], 


(69) 


We have the following cases: 


1) If /(Amax) > 0, or equivalently, by using (j^ and (|3^, > Amax, 

then An(s) = Amax- It follows that the condition An(s) < Xmp is violated. Note that, for this 
case to occur, I4_i(s + 1) — I4_i(s) > 0 must necessarily hold. 
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2 ) If /(Amin) > 0, or equivalently ^ Amin, then An(s) = Amin- It follows 


that the eondition An(s) < Xmp is satisfied. Using ( |36[ ), this ease is equivalent to 

■ 1 


Ki-i(-s) — Ki_i(s + 1) > 


^In I 

i-p \p 


log 2 (e) = 7 . 


(70) 


3) Otherwise, An(s) is the unique solution of f{x) = 0. Using (68), we thus obtain 

An(s) = (71) 

In this ease, the eondition A„ (s) < Xmp is satisfied if and only if 14_i(s + 1) — 14_i(s) < 0. 


By eombining eases 1 , 2 and 3 , we obtain that the eondition stated in (67) is equivalent to 

I4i-i(s) — Vn-i{s + 1) > 0. (72) 

We prove (|7^ by induetion on n, thus implying ([67]). Trivially, (72) holds for n = 1, sinee 


Vo(s) = 0,Vs. Now, let n > 1 and assume that the eondition (72) holds under sueh n. We show 


that (72) implies 


I7n(s) — Vn{s + 1) > 0, 


(73) 


proving the induetion step. Sinee ease 1 eannot hold under the induetion hypothesis, by eombining 
oases 2 and 3 , An(s) oan be expressed as 


A„(s) = max{Amin,2 


(74) 


Therefore, by reordering the terms and using (66), we obtain 

I7n(s) — Vn{s + 1) = l(Xn{s), s) — I(Xn{s + 1), S + 1) 

+ (I4i_i(s) — Vn-l{s + 1))(1 — 5/i(s + 1) — (5A(s)An('S)) 

+ 6^{s){Vn-i{s — 1) — 14i_i(s)) + 5A(s + l)A„(s + l)(14i_i(s + 1) — Vn-l{s + 2)) 

= 5'(I/i-l(-S — 1), I4i_i(s), Vn-l{s + 1), Vn-l{s + 2)). (75) 


Therefore, the induetion step 14(s)—14(s+l) > 0 is equivalent to giVn-i^s—l), 14_i(s), 14_i(s+ 
l),14_i(s + 2)) > 0. Note that giVn-ii^s - 1), 14_i(s), 14-i(s + l),14-i(s + 2)) is a non- 
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decreasing function of Vn-i{s — 1). Since i4_i(s — 1) > i4_i(s) from (72), we thus obtain 

Ki(s) — Vn{s + 1) = g{Vn-l{s — 1), i4i_i(s), Vn-l{s + 1), Vn-l{s + 2)) 

^ fi'(Ki-i(s), i4i_i(s), Vn-i{s + 1), Vn-i{s + 2 )) = /(A„(s), s) — I (A„(s + 1), S + 1) 

+ (V^n-l('S) — Vn-l{s + 1))(1 — (5/i(s + 1) — 574 (s)A„(s)) 

+ 5y4(s + l)A„(s + l)(i4i-i(s + 1) — Vn-\{s + 2 )). (76) 

We now minimize g{Vn-i{s),Vn-i{s),Vn-i{s + l),i4-i('S + 2)) with respect to i4_i(s), and 
use the induction hypothesis i4_i(s) > i4_i(s + 1). We distinguish the two cases Vn-i{s) — 
( 4-1 (s + 1 ) > 7 and 7 > (4-i(s) - (4-i(s + 1 ) > 0 below. 


a) Case (4_i(s) — (4-i(s + 1) > 7-' In this case, (74) yields A„(s) = Amm, and therefore. 


substituting in ( |76| ), 

5'(I4-i(s), 14_i(s), 14_i(s + 1), 14-i(’S + 2)) 

= (I4-i(s) — I4-i(’S + 1))(1 — Sg,{s + 1) — 54(s)Amin) — /(A„(s + 1), s + 1) 

+ + l)A„(s + l)(I4_i(s + 1) — 14-i(s + 2)). (77) 

The derivative of g{Vn-i{s), 14-i(s), 14-i(5+l), 14_i(s+2)) with respect to 14_i(s) is given by 
dg{v,v, Vn-i{s + 1), ■l4-i(s + 2)) 


du 


— 1 — (5/i(s + 1) — 54(s)Amin > 0, 


(78) 


V=Vn-\{s) 


where we have used 


b) 7 > 14-i(s)—14_i(s+l) > 0.- In this case, (74) yields An(s) = 2 '5['7-i(s) 


Substituting this expression in (76), we obtain 

dg{v,v, 14-i(s + 1), I4-i(s + 2)) 


du 


= 1 — 5/i(s + 1) — 54(s)A„(s) 


V=Vn-l{s) 

+ <5"Al(s)2-'5[^"-i(^)-'""-i(^+i)lAMpln2(K-i(s) - I4-i(s + 1))' 

- /'(A4s),s)2-''[^--(*)-'"'^-(*+')l Wln25[K-i(s) - I4-i(s + 1)]. 


Now, from ( [681 ) and ( j^ , we have that 

7^(An(s),s) = 4(s)()[14_i(s) — 14_i(s + 1)], 


and therefore 

dg{v, V, Vn-i{s + 1), ■l4-i(s + 2)) 


du 


= 1 — 6fi{s + 1) — 6A{s)Xn{s) > 0. 


(79) 


(80) 


(81) 


v=V„-i{s) 
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In both cases I4_i(s) — Vn-i{s + 1) > 7 and 7 > I4_i(s) — I4_i(s + 1) > 0, we have that 
g{Vn-i{s), I4-i(-s), I4-i(s + l), I4-i(s + 2)) is a non-deereasing funetion of I4-i(s), minimized 


by I4-i(s) = Vn-i{s -I- 1 ), yielding An(s) = Xmp- From (76), we thus obtain 


Vn{s) — Vn{s -f 1) — giVn-l^S — 1), Vn-l{s + 1), Vn-l{s + 2)) 


^ 5'(Ki-i(’S -f 1), Vn-l{s + 1), Vn-l{s -f 1), Vn-l{s + 2)) 


— I{Xmpi ■s) ~ -^(An(s -f 1), S 1) -f (5y4(s + l)A„(s + l)(I4i_i(s -f 1) — Vn-l{s + 2)) 

^ I(XmP, s) — l(Xn{s + 1), s + 1), (82) 


where in the last inequality we have used the induetion hypothesis 14 -i(s + l) — 14 -i(s + 2) > 0 . 
Finally, using (|3T|) and the faet that ^4(0) = 1 from Assumption we obtain 

-^(Amp; s) — /(A„(s + 1 ), s -f 1 ) = A[s)I(\mpi 0) — ^(s + l)-f(A„(s + 1 ), 0 ) 

= (A(s) — A(s + 1))I{Xmp, 0 ) + A(s + 1 ) [/(Amp, 0)-/(A„(s + l),0)] >0, (83) 

where we have used the definition of Xmp in ( [3^ and the faet that A(s) > A(s + 1 ) from 
Assumption Therefore, 14(s) — 14(s -f 1) > 0, proving the induetion step and the theorem. 


Appendix E: Proof of Proposition [5] 


In the policy evaluation step [ D|, given the policy A 1*1 at the beginning of the fih stage of the 
algorithm, we evaluate the value funetion pW(s) over the states s G S. Let be the aehievable 


rate under poliey A^. The value funetion under A^ is the solution of [13| 

pW(s)- P(^fc+i=si|^fc=s, AW(s))pW(si)=/(AW(s),s) - Vs G 5 (84) 


with r;l*q0) = 0. Equivalently, using (25), we obtain 

pW( 0 ) = 0 , 

p^*^(s) — (5/i(s)p^*^(s — 1) — (5A(s)Af*^(s)pf*^(s + 1) (85) 

- [1 - 5 / 2 (s) - (5A(s)AW(s)]pW(s) = J(AW(s),s) - Vs G 5, 

Let 77^(0) = 0 and /7W(s) = 5[pW(s) — pW(s — 1)], Vs > 0. Then, the reeursive expression 
(34) direetly follows by rewriting ( 86 ) in terms of 77W. 


Januaiy 19, 2015 


DRAFT 









29 


In the policy improvement step, an improved policy is determined as 

A[*+d(s) = argmax /(x, s)+^ P(S'fc+i=si|S'fe=s,AW(s))nW(si), (86) 

a:S[Aniin,Vjv./p] siScS 

where we have used Theorem to restrict the optimization over [Amin, Amp], and, using (25), 

A^*’''^l(s)= argmax I{x, s) + — 1) + + 1) + [1—M(s)a;]p^*^(s) 


= argmax /(x, s) + 74(s)a;Pf*^(s + 1). 

[-^min 5-^Af p] 


(87) 


Since the objective function in ( [87[ ) is concave in x, we have the following cases, yielding ( |35] ): 

1) I'(Xmp,s) + 74(s)pW(s + 1) > 0, where I' (x, s) is given by ( | 68 | ), or equivalently, using the 
expression of Amp in (36), pW(s + 1) > 0. In this case, the objective function in (87) is an 
increasing function of x, hence the optimal is A[*’''^l(s) = Amp- 

2) /'(Amin, s)+A(s)pW(s+l) < 0, or equivalently pW(s+l)< log 2 (e)+ log 2 (p) In this case, 
the objective function in (87) is a decreasing function of x, hence the optimal is A[*+^](s) = Amin- 

3) Otherwise, the optimal A[*+^l(s) is the unique x G [Amin, Amp] such that I' (x, s)+y4(s)pW(s + 


1 ) = 0. Using the expression of I'{x,s) (68) and of Amp in (36), we obtain 

e 

The specific form of the policy iteration Algorithm is thus proved. 


( 88 ) 
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